Add an optimization doc on TPU #21155
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new documentation page with optimization tips for using vLLM on TPUs. The content is valuable and well-structured. I've identified a few issues, mainly related to broken or fragile links and a missing image, which could impact the user experience. I've also found a typo in a command-line argument that could cause issues for users who copy-paste it. Please see my detailed comments.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
@Chenyaaang @yaochengji ptal |
docs/configuration/tpu/README.md
Outdated
|
|
||
| Although the first compilation can take some time, for all subsequent server launches, vLLM can load these graphs directly from the cache, eliminating the compilation time for future runs. | ||
|
|
||
| Use `VLLM_XLA_CACHE_PATH` environment variable to write to shareable storage for future launches. |
There was a problem hiding this comment.
Do we need to mention that there's actually a issue in persistent compilation cache?
There was a problem hiding this comment.
Thanks for the docs contribution!
I have some general style comments initially:
- Please fix the
pre-commitandDCOchecks - Please avoid using bold text in the headings
- Some of the headings are quite long might not render how it was intended
- In the navigation bar this page is nested in a TPU directory and you only see the title of the page if you try and look inside it
docs/configuration/tpu/README.md
Outdated
There was a problem hiding this comment.
Why docs/configuration/tpu/README.md instead of docs/configuration/tpu.md
docs/configuration/tpu/README.md
Outdated
| #### **SPMD** | ||
| More details to come. | ||
|
|
||
| #### Want us to cover something that isn't listed here? Open up an issue please and cite this doc. We'd love to hear your questions or tips. |
There was a problem hiding this comment.
This should't be a heading, the whole thing appears in the ToC
docs/configuration/tpu/README.md
Outdated
|
|
||
| ### **Get started** | ||
|
|
||
| Looking for setup and installation instructions? Find them [here](https://docs.vllm.ai/en/latest/getting_started/installation/google_tpu.html). |
There was a problem hiding this comment.
Please use relative md links for internal docs references (http links will break in future releaes)
docs/configuration/tpu/README.md
Outdated
|
|
||
| #### **Profiling** | ||
|
|
||
| The auto-tuner provides a profile of optimized configurations as its final step. However, interpreting this profile can be challenging for new users. We plan to expand this section in the future with more detailed guidance. In the meantime, you can learn how to collect a TPU profile using vLLM's native profiling tools [here](https://docs.vllm.ai/en/latest/examples/offline_inference/profiling_tpu.html). This profile can provide valuable insights into your workload's performance. |
There was a problem hiding this comment.
Please use relative md links for internal docs references (http links will break in future releaes)
docs/configuration/tpu/README.md
Outdated
| ### **If possible, use the precision that matches the chip’s hardware acceleration** | ||
|
|
||
| - v5e has int4/int8 hardware acceleration in the MXU | ||
| - v6e has int4/int8 hardware acceleration in the MXU | ||
|
|
||
| ### **Don't set TP to be less than the number of chips on a single-host deployment** |
There was a problem hiding this comment.
These headings are also quite long
There was a problem hiding this comment.
LGTM!
Could you fix the pre-commit and DCO? https://docs.vllm.ai/en/latest/contributing/index.html#code-quality
Creating a place in the docs for TPU optimization tips, WIP